Learning Matchable Image Transformations for Long-term Metric Visual Localization

Long-term metric self-localization is an essential capability of autonomous mobile robots, but remains challenging for vision-based systems due to appearance changes caused by lighting, weather, or seasonal variations. While experience-based mapping has proven to be an effective technique for bridging the `appearance gap,' the number of experiences required for reliable metric localization over days or months can be very large, and methods for reducing the necessary number of experiences are needed for this approach to scale. Taking inspiration from color constancy theory, we learn a nonlinear RGB-to-grayscale mapping that explicitly maximizes the number of inlier feature matches for images captured under different lighting and weather conditions, and use it as a pre-processing step in a conventional single-experience localization pipeline to improve its robustness to appearance change. We train this mapping by approximating the target non-differentiable localization pipeline with a deep neural network, and find that incorporating a learned low-dimensional context feature can further improve cross-appearance feature matching. Using synthetic and real-world datasets, we demonstrate substantial improvements in localization performance across day-night cycles, enabling continuous metric localization over a 30-hour period using a single mapping experience, and allowing experience-based localization to scale to long deployments with dramatically reduced data requirements.Comment: In IEEE Robotics and Automation Letters (RA-L) and presented at the IEEE International Conference on Robotics and Automation (ICRA'20), Paris, France, May 31-June 4, 202

Vision-based Categorization of Upper Body Motion Impairments and Post-stroke Motion Synergies

Upper body motion impairment is a common after-effect of a stroke. We are developing a novel multisensory therapy system that makes use of augmented feedback and is expected to enhance rehabilitation outcomes. The first phase of the project involved developing algorithms to automatically differentiate between normal and impaired upper body motions. Seven healthy subjects performed two types scripted actions: (a) elbow flexion and extension and (b) reaching over via elbow flexion and shoulder flexion and adduction. Each action was repeated 10 times as each participant felt most comfortable and also 10 times simulating a common post-stroke impaired motion according to the literature. Principal component analysis was applied to the upper body trajectories during each action, as observed with a Kinect sensor, to extract the dominant modes of motion. Three classification algorithms and up to three motion modes were examined in order to distinguish between normal and impaired motions. A statistical analysis of the Kinect skeletal tracking data vs. manual annotation confirmed a significant bias in the tracking of an elevated shoulder joint. Despite this bias, leave-one-subject-out cross validation experiments confirmed the effectiveness of the proposed methods in detecting impaired motions (accuracy > 95%), even when the impairment involved elevating a shoulder. A single, most dominant motion mode provided sufficient information for the binary classification task. The high accuracy rates validate the use of vision-based pose tracking technologies in identifying motion deficiencies